Methods

From Research Design to Methods: Explaining the How and Why

Bogdan G. Popescu

John Cabot University

Methods Section of the Article

What to include

The methods section should include answers to the following questions:

  • How did you do your research?
  • What was the research design?
  • Was it quantitative or quantitative?

Introduction

The Research Design

The Research Design = the overall plan to answer a research question.

It forms the backbone of the methods (tools) sections of a paper.

  • What data did you use and how was it collected?
  • What are your key variables and how were they measured?
  • What analytical methods did you apply?

A good research design tries to ensure high internal and external validity

The Research Design

Internal Validity

Internal Validity - the extent to which we can be confident that the independent (causal) variable produced the observed effect.

  • Is the independent variable responsible for variation in the dependent variable?
  • What other possible causes might there be for the relationship between the variables?
  • Could something elsehave been responsible for the variation in the dependent variable?
  • Could there be confounding factors?

The Research Design

External Validity

External Validity - the extent to which the results from a study can be generalized beyond the particular study

  • Can you generalize your findings?
  • Are your conclusions likely to apply more widely?

Methods

Research methods = data collection + data analysis.

Approaches:

  • Quantitative approaches - usually good at making generalizations: large-N
  • Qualitative approaches - good at investigate these hard-to-define concepts and hard-to-reach populations: small-N

Data sources include:

  • already existing quantitative data
  • experiments
  • surveys
  • interviews or focus groups
  • comparative research
  • archival data and documentary records (e.g., speeches, policy documents)

Units of analysis (observations) can include people, countries, organizations, texts, etc.

Types of Research Designs

General Overview

Research designs can be classified in various ways. Common types include:

  • Comparative designs – comparisons across countries, regions, or groups using observational data
  • Exploratory (e.g., fieldwork, open-ended interviews)
  • Explanatory/Causal (e.g., experiments, quasi-experiments, regression designs)
  • Cross-sectional vs. longitudinal
  • Quantitative vs. qualitative

Types of Research Designs

Focus of This Course

The ones that we will focus on are:

1.Comparative Analysis - qualitative
2.Experimental Designs - quantitative
3.Quasi-natural Experiments - quantitative

Comparative Analysis

Case Study: Intro

A case study focuses on the analysis on one country, event, or organization.

It must be situated comparatively to matter beyond itself.

Case studies:

  • apply existing theory to new contexts;
  • examine exceptions to the rule
  • generate new theory
  • have to explain how they are applicable to other contexts
  • explore a causal process or mechanisms (process tracing)

Comparative Analysis

Case Study: Data

Case studies may use

  • interviews
  • surveys
  • ethnography
  • focus groups
  • documents: policy documents and speeches

Comparative Analysis

Case Study: Examples

Voting behavior in Britain

Public Attitudes towards the environment in Germany

Public attitudes towards immigrants and ethnic minorities in the Netherlands.

Comparative Analysis

Comparative Designs: Intro

Comparative designs can be used:

  • to apply existing theory to new cases;
  • to develop new theory or hypotheses
  • to test theory

Small-N studies are sometimes used for:

Process Tracing - uncover causal paths and mechanisms - assess specific mechanisms identified in theories.

Comparative Analysis

Comparative Designs: Small-N Comparison

Small-N Comparison involves the the comparison of two or more cases

  • they offer detailed in-depth analysis of the case study,
  • they provide greater scope for contextualization.

They can be based on the most similar research design:

  • selecting countries that share many (theoretically) important characteristics, but differ in one crucial respect (related to the hypothesis of interest).

They can be based on the most dissimilar research design:

  • cases that are different in most respects and only similar on the key explanatory variable of interest.

Comparative Analysis

Comparative Designs: Small-N Comparison

Examples:

  • Michael Lewis-Beck’s (1986) economic voting in Britain, France, Germany, and Italy
  • Seymour Martin Lipset’s (1959) study of the social requisites of democracy in Europe and South America;
  • Thelda Skocpol’s (1979) study of revolution in Russia, France, and China;

Types of Research Designs

Experimental Designs

Most rigorous way to test if X causes Y. Why?

  • You control who gets treatment vs. control
  • Randomization eliminates confounders

Basic Steps

  1. Split into treatment and control groups
  2. Randomly assign participants
  3. Measure outcome before the intervention (pre-test)
  4. Apply the treatment
  5. Measure outcome again (post-test)

Types of Research Designs

Natural Experiments

You don’t control the treatment — but nature helps you out.

Key designs:

  • differences-in-differences
  • regression-discontinuity design

These methods try to mimic experiments, even without full control.

Types of Research Designs

Natural Experiments: Assumptions

Quasi-experiments need:

  • Variation in the causal variable that’s independent of confounders
  • That variation must be as good as random

Hard to guarantee in practice — strong designs are needed.

Note: These designs often have high internal validity, but limited generalizability (lower external validity)

Methods

Using the Model Article

Let us examine the model article that we identified

Depending on the method you use you will have to answer different questions

Methods

To Do 1

Get your draft under the methods section the answer to the following questions:

  • what data are you using
  • where is the data from
  • how novel is the data
  • how does the data help you answer the research question

Methods

To Do 2

Get your draft type under the methods section the answer to the following questions:

  • how are you analyzing the data
  • how are you interpreting your data
  • are there any limitations to your data

Methods: Quantitative Methods

To Do 3

If you use a quantitative research design, you should also answer these questions

  • What variables did you analyze?
  • Why did you choose those variables?